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Evaluation and feedback are important in any educational 
debate situation. We assume debaters are aided in learning 
skills through accurate feedback and, in most instances, the 
critics communicate this feedback only through the use of a \^ 
written ballot. It is hoped that the information provided by 
ballots will aid the student in evaluating and thus improv- 
ing his performance. 

As everyone knows, we do not always put a great deal of 
faith in the accuracy of these comments. It is not at all 
uncommon for debaters, and for that matter their coaches, to 
question, the credibility of judges/ particularly the so- 
called " iiurvqualified judge." 

Tiside from the question of whether competitive debate 
should usfe the professional or nonprofessional judge, most 
tournament directors are necessarily in the position' of using 
a large body of nonprofessionals (i.e., non-coaches) as judges. 
■ Research to date has found that the professional and non- 
professional^ judges differ in several ways. A number of 
studies have tended to show that while coaches and non-coaches 



almost always reach the same deci^ionsf they aire not for the 
same reasons,"*' In general^ professional judges found "argu- 
ment"' to'Ee the major factor in d^^termining ^ winner, while 
nonprofessional judges found "deli\rery" and delivery-related 
areas most influential.^ Little research/ however, has 
focused on the differences in fee<aipaclc provi<ae<a by the written 
ballot. Specifically, would significant di£5ei:ences be 
expected between professional and nonprofessional judges in: 
tendency to award the decision to one side, ^inount of quality 
points aW^rdeca, team ratings ass:igned, and ainount of comments 
or feedback on ballots. In othej: words, what variations in 
feedback can be expected from professional versus nonprofes- 
sional juciges"? 

This stu^y focuses on a comp^jrison of professional and 
nonprofessional judges in providiiig feedback by means of the ^ 
written ballot. Such an analysis has several implications. 
Since most debaters engage in intercollegiate competition, 
learning how one will be judged atid how to better interpret 
the ballot feedback will be useful. Also, beo^use decisions 
on who will advance to elimination rounds of tournaments are 
often closer the variations of pjTQifessional ^nd nonprofessional 
judges may significantly affect it. It couia make possible, 
then, greater fairness in assigni^ig judges. 



Method 

The data for this study wer^ gathered in th^* six prelimi- 
nary rounds at the Big Sky Interoollegiate Fcjrensic Tournament 



hosted by the University of Montana, April, 1971. The tourna- 
ment is an open-invitational and attracts competition from 
approximately thirty colleges largely in a five-state North- 
west region. The debates utilized the Parliamentary style 
format^ on the national debate topic, "Resolved: That the 
federal government should establish a program of wage and 
price controls." Half of the debates were judged by profes- 
sional judges and half were judged by nonprofessional judges. 
All judges used Form C of the American Forensic Association 
ballot and were given an instruction sheet describing the for- 
mat of the debate and defining the six criteria contained on 
the ballot. 

The professional judge:, were judges provided by the guest 
schools. This group consisted largely of forensic coaches and 
graduate forensic assistants. The nonprofessional judges were 
volunteer adults obtained from the surrounding community . 
This group consisted largely of business and professional peo- 
ple and faculty meiTibers from various departments in the uni- 
versity. None of the jxidges were paid. 

The professional judges could generally be categorized 
as being experienced both in the theory and practice of debate 
and in judging the national debate topic. The nonprofessional 
judges lacked one or both of these qualifications. 

The judges were arbitrarily assigned to rounds with no 
conscious attempt to select judges for particular debates. 
Random assignment in its purest sense could not be employed/ 
however, since it was impossible to control the judging times 



failed to fill out some scale on the ballot, the data were 
omitted in the analysis of results concerning that particular 
scale. 



Results 

The results are presented as answers to particular 
research questions asked by the investigators: 

1. Do professional and nonp r ofessional judges tend to award 
the decision to one side more often than the other ? 

Table I presents the number of ballots awarded to the 
affirmative or negative sides by the professional and non- 
professional judges. 

Table I 

Frequency of Wins Awarded to Sides by Judges 



- Professional Nonprofessional 
' Judges Judges 

Affirmative win 58 45 

Negative win 41 52 
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The data from Tabic I were analyzed using the Chi-Square 
Test corrected for continuity^ yielding a nonsignificant 

2.45. The data were further subdivided to allow analy- 
sis of only the ballots from the senior division (TC = 2.31) 
and junior division ( >^ = 2.14)\ both of which were nonsig- 
nificant. This analysis revealed that there were no signifi- 
cant differences in the sides (affirmative or negative) 
awarded the decision by professional versus nonprofessional 
judges. 

2 . Do professional and nonprofessional judges differ in the 
number of quality points awarded to the individual speak- 
ers m the debates ? 

Most coaches and debaters are familiar with the 30-point 
rating boxes provided on the AFA ballot, form to rate each 
speaker on a scale from one to five points ("poor" to "super- 
ior") on each of six criteria (analysis, reasoning, evidence, 
organization, refutation, and delivery) . Since these quality 
points or speaker points are often used to break ties to 
determine finalists among teams with identical win-loss records, 
it is important to know whether the type of judge employed in 
the debate biases these ratings. Table II presents the mean 
speaker ratings by the professional and nonprofessional 
judges and t-tests of the differences between the means for 
each division. 

The results of the data in Table II showed that nonpro- 
fessional judges tended to award significantly higher quality 
points to the individual speakers in both divisions than did 
the professional judges. 



Table II 

Mean Quality Points Awarded to Speakers in Each Division 



t 



Source of Professional Nonprofessional 



Speaker Ratings 



Judges • Judges t 



sr. Div. only 21.0 23.0 3.96* 

n=260 



Jr. Div. only 
n=528 

Both Divisions 
(combined) 



20.2 21.6 4.21* 

20.5 22.1 5.49* 



< .05- 



3. Do professional and nonprofessional judges differ in the 
quality ratin g assigned t o th e teams ? 

The AFA ballot, ^prm C, also provides a simple five-point 
scale (ranging from "poor" to "superior")., for rating the 
quality of each of the two teams in the debate. In the tour- 
nament, 393 such team ratings were reported on the ballots. 
Professional judges awarded a mean team rating of 3.41 and 
nonprofessional judges awarded a mean team rating of 3.66. 
Although the mean raf igs of the two types of judges differed 
by only a quarter of a point, this difference was statisti- 
cally significant (t=3v34, p < .05), indicating that the 
nonprofessional judges awarded' significantly higher quality 
ratings to teams than the professional judges. 

4. DO professional and nonprofessi onal judges differ in the 
Snoiint of written feedback made on the ballot .? 

The AFA ballot, form C, provides spaces for written 



comments regarding each of the four speakers in the debate. 
Other than the actual win- loss judgment, these written cri- 
tiques have the potential to be the most speicific and helpful 
part of the feedback from the judge to the debaters available 
on the ballot. Several analyses were made of the ballots from 
the Big Sky tournament to determine the extent to which the 
professional and nonprofessional judges utilized this oppor- 
tunity to comm\inicate in writing to the debaters. 

Perhaps nothing is more frustrating to a debater or coach 
than to read a decision of win or lose on a ballot but to 
receive no explanation or comments from the judge to indicate 
why he made such a decision or what he liked or disliked about 
the debaters' performances. A simple analysis was performed 
to determine whether professional and nonprofessional judges 
differed in the tendency to hand in blank, no comment ballots. 
None of the 99 ballots provided by the professional judges 
were completely blank but 22 of the 98 ballots provided by 
nonprofessional judges were completely blank in the spaces 
provided for comments. Table III shows the number of ballots 
coming from professional and nonprofessional judges which 
were lacking in feedback for at least one debater versus 
those in which comments were provided for all four debaters. 

The statistical analysis of the data in Table III showed 
a highly significant tendency in which professional judges 
more often provided written feedback to all four debaters than 
•did the nonprofessional judges (X = 25. 49; p < . 001) • 
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Table III 

Frequency of Ballots from Professional Versus 
Nonprofessional Judges Containing Feedback 
Versus No Feedback for Debaters 



Description of Professional Nonprofessional 

Feedback Judges Judges 



Comments for all 
four debaters 94 .. 64 

Blank ballot or 
no comment for at 5 34 

least one debater 



A random sample of 25 ballots from the professional 
judges and 25 ballots from the nonprofessional judges was 
taken to allow a word count of the written comments to fur- 
ther describe the amount of feedback provided by the judges. 
The mean number of words per ballot written by professional 
judges was 148 words (ranging from 46 words to 541 words), 
while the nonprofessional judges wrote a mean of 57 words per 
ballot (ranging from zero to 154 words). It appeared that 
the professional judges provided nearly three times as inauch 
written feedback as the nonprofessional judges. Indeed, the 
lengthiest set of comments from the nonprofessional judges 
(154 words) was only slightly higher than the average of the 
professional judges (148 words) . 

Of course, the presence or absence of comments is only 
a crude measure of the quality of feedback provided by the ' 
judges. However, in reading the comments in the sample of 
50 ballots used in the word count analysis , the authors agreed 
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that the professional judges tended more often to provide sub- 
stantive comiBents and critiques on the cases and debate tech- 
niques employed by the debaters^ whereas the nonsubstantive^ 
evaluative comments such as "good debate" and "you have an 
. excellent voice" tended more often to come from the nonprofes- 
sional judges. Clearly, the professional judges provided a ? 
greater amount of written feedback to the debaters in contrast 
to the nonprofessional judges* 

Discussion 

From the results of this study, a general picture of the 
comparison and contrast of professional and nonprofessional 
judges emerged. The composite of the professional judge 
appeared to be an individual who had no particular preference 
for the side of the question being argued; who tended to be 
somewhat stingy in rating the individuals and teams; and who 
tended to provide a substantial amount of written feedback 
on the ballot. The composite nonprofessional judge appeared 
to be an individual who also had no particular preference for 
the side of the question being argued; who tended to be some- 
what generous in rating the individual speakers and teams; and 
who tended to provide little or no written feedback on the 
ballot. 

In tournaments where both professional and nonprofessional 
judges are used, an attempt should be made to give each team a 
proportionate number of each type of judge to offset the dif- 
;ferences in rating behavior if such ratings are ever to be 
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used as tie-breakers for finalist selections. If such a pro- 
cedure is not possible, the first criterion for breakina ties 
should be based on a system such as median ratings rather than 
mean or total points, to offset the bias of a generous or 
stingy judge. 

There was no evidence in this study to suggest that the 
nonprofessional judges were incompetent to judge the debates. 
The stereotyped image of the conservative townsperson, unwill- 
ing to give a fair hearing to the radical affirmative proposal; 
did not appear in the data. At most, the results suggested 
that the nonprofessional judges may have been somewhat 
insecure about their duties, thus tending to avoid writing 
comments on the ballots which would reveal the bases of their 
decisions. The practice of using nonprofessional judges to 
judge "unimportant" debates should probably not be carried to 
extremes, therefore, since the debaters In .such debates are 
usually those who could profit most from the extensive and 
substantive critique of a trained professional judge. 
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